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1,  Introduction 

Many  interesting  aspects  of  informetion  theory  can  be  illustrated  by 
stv.dyi-ng  discrete  channels  with  small  input  and  output  alphabets.  In  fact, 
much  important  work  has  been  done  on  the  simplest  of  all  channels,  the 
(memoryless)  binary  symmetric  channel  (BSC).  We  refer  to  Elias'  study  of 
the  way  in  which  rate  and  probability  of  error  in  the  BSC  depend  on  the  length 
of  the  blocks  in  which  the  coding  is  done  [l] .  In  the  present  article,  we 
shall  be  concerned  exclusively  with  asymptotic  properties  of  channels,  i.e. 
properties  which  are  based  on  the  assumption  that  coding  is  done  in  arbitrarily 
long  blocks.  In  this  study,  we  shall  sometimes  consider  temarj'-  and  higher 
order  channels  as  well  as  binary  channels.  In  fact,  the  binary  channel  is 
too  simple  to  be  representative  of  the  general  discrete  channel,  as  can  be 
seen  from  the  fact  that  neither  input  letter  can  be  suppressed  without 
destroying  its  information  rate  . 

2,  Information  sources  and  information  rate 

The  first  concept  of  interest  in  information  theory  is  that  of  an 
information  source,  i.e.  a  device  which  generates  a  random  sequence  of  letters 
from  some  alphabetj  the  random  sequence  then  serves  as  the  input  to  a  channel 


X  . 

m 


(see  Section  3).  Let  the  input  alphabet  consist  of  the  m  letters  x,,  .. 
We  shall  confine  our  attention  to  independent  sources,  for  which  the  probability 
P.,  1  <  i  <  m,  that  the  input  letter  x.  is  emitted  at  time  t  is  statistically 
independent  of  which  letters  were  emitted  at  times  prior  to  t  and  of  which 


4* 

Even  in  the  most  asymmetric  binary  channel  one  does  not  have  to  use  an  input 
symbol  more  often  than  63  per  cent  (or  less  often  than  37  per  cent)  of  the  time 
to  achieve  capacity  (see  Section  $) , 
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letters  will  be  emittRd  at  -times  si^bsequent  to  t.       Thus,   the  probability  of 
a  sequence  in  which  x.    appears  N.    tines,   1  <  i  <  m,   is  just 

N.  N_  N 

According  to  Shannon   [2],   it  is  particularly  meaningful  and  fruitful  in 
studying  information  sources  to  introduce  the  concept  of  the  entropy  or 
information  associated  with  a  source j   for  a  source  S  of  the  type  under 
consideration,   this  quantity  is  defined  by 


m 
(1)  H(S)  =  -  J~     P,   log  P. 


X=l 


The  base  to  which  the  logarithm  is  taken  is  conventionally  chosen  to  be  2, 
in  which  case   (l)   is   said  to  give  the  number  of  bits  of  information  per  source 
letter  (see  rem.arks   at  the  end  of  this  section);  whenever  we  write  log  we  shall 
mean  log_.     Henceforth,  it  will  be  assumed  that  the  source  emits  one  letter 
per  second  (and  that  the  channel  accepts  one  letter  per  second).     With  this 
convention,    (1)  gives  either  the  entropy  of  the  source  in  bits/sjonbol  or  the 
information  rate  of  the  source  in  bits/second.     This  contention  allows  us  to  use 
the  terms  entropy  (or  information)   and  rate  interchangeably;   th?  adjustment 
needed  in  case  the  source  emits  one  letter  every  T  seconds  is  obvious. 

We  shall  not  linger  on  the  derivation  of  (1)   from  a  set  of  properties 
which  it  seems  reasonable  to  expect  information  to  have.     Such  derivations  are 
given  in  detail  by  Shannon   [2] ,  Khinchin   [3]   and  Faddeyev   [li]  .     An  important 
property  of  H(s)  is  tha.t  it  vanishes  if  the  source  can  emit  only  one  of  the 
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letters  x,  ,    ....  x   ,   p.nd   tekes  its  maximum  value   (log  m)  when  the  source  emits 
I'm' 

all  01   the  letters  x,  ,    ....  x     with  equ?].  probability.      (Tlnis   is  in  accord 

X  m 

with  intuitive  ideas  of  information.)     Another  important  property  of  H(s) 
is  the  fact   that  although  S   can  emit  m     possible  sequences  of  length  M,   one  of  a 
much  smaller  set  of  2  sequences  is  veiy  likely  to  occur  if  M  is  very 

large.      (An  exceptional   case  occurs   if  all  m  source  letters  are  equally  likely; 
then  H(S)   =  log  m     and  2     ^^"^   =  m  . )     This  so-called  asymptotJc  equipartition 
property  is   fundamental  in  info rniat ion  theorj*  and  can  be  demonstrated  for 
much  more   general   sources  than  those  studied  here    (in  fact  for  any  stationary 
ergodic  source;   see  McMi3.1an   [?] ,  Khinchin  [3]).     Finally,  we  remark  that  tl:ie 
first  of  Shannon's  two  coding  theorems   (the  so-called   noiseless   coding  theorem) 
consists   in  showing  that  precisely  H(S)  binary  digits   (bits)   per  symbol  are 
needed  to  noiselessly  encode  the  output  of  S  into  binary  digits;  in  general 
it  takes  code  blocks  of  infinite  length  to  effect  this  encoding.     The  noiseless 
coding  theorem  gives  theoretical   justification  for  measuring  H(s)   in  bits/symbol 
(or  bits/second), 

3.     Channels  and  mutual  inf ormati on  rate 

A  discrete  mxn  nemoryless   (information)   channel  is  a  probabilistic 
device  which  accepts  any  of  m  possible   input  (or  "  transmitted")    letters 
X, ,    ...,  x^  and  emits  any  of  n  possible  output  (or   "  received"  )  letters 
y. ,    ,..,  y  ,  in  accordance  with  the  following  rules: 

1)  For  every  input  letter  x.   and  output  letter  y.  there  is  a 
definite  number  p. .,  0  <  P- ^  <  1>  which  represents  the  probability  that  if  x. 
is  transmitted,  y     is  received. 


I.e. 


-  U  - 


2)  '^'■erj'"  input  letter  gives  rise  to  at  least  one  output  letter. 


ij 


1. 


3)  The  response  of  the  device  to  any  input  letter  x.    is 
statistically  independent  of  its  response  to  pny  past  or  future  letter.     In 
Section  8  we  shall  abolish  this  requirement  and  consider  a  special  kind  of 
discrete  channel  with  memory. 

A  convenient  representation  of  such  a  discrete  channel  is  as  an  m>*n 
channel  matrix  C,   i.e.  as  the  mxn  rectangular  array  of  numbers 


(2) 


PlI  P12  •••  Pin 
P2I  P22  •••  P2n 


Pml  Pm2  •••  Pmn 


(We  shall  use  C  to  denote  either  a  channel  or  the  associated  matrix.)  By 
1)  and  2)  each  p, .  lies  between  0  and  1,  and  the  sum  of  every  row  of  the  matrix 
(2)  is  unity;  these  properties  are  summarized  by  calling  C  a  stochastic  matrix. 
The  study  of  stochastic  matrices  has  received  a  great  deal  of  attention  in  the 
mathematical  literature,  especially  in  connection  with  the  theory  of  Markov 
chains  (see  Feller  [6]).  We  note  that  the  case  of  a  noiseless  channel  corresponds 
to  the  case  where  each  row  contains  one  1  and(m  -  1)0 's,  while  each  column 
contains  one  1  and  61  -  1)  0 '  s . 


Suppose  now  that  the  input  letter  x. ,  1  <  i  <  m,  is  used  with  probability 
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P.   and  define  p(i,j),   the  joint  probability  that  x.    is  emitted  and  y.  is 
received,  by  the  relation 


P(i,j)  =  P^  P^j 


Then  the  probability  that  y .,  1  ^  J  ^  n,   is  received,   regardless  of  which  x. 
is  transmitted,   is  given  by 


m 


'r 


S^'^ 


,J) 


It  is  convenient  to  represent  both  the  numbers  P. ,  1  <  i  <  m,  and  the  numbers 
P.,  1  ^  J  <  i^j  ss  the  components  of  corresponding  column  vectors  P  and  P  ,  i.e. 


m 


P'  = 


^2 


P' 
n 


We  now  give  a  fundamental  definition  introduced  by  Shannon  [2]  :     The 

average  mutual  information  rate  (or  sipiply  the  rate)  of  the  cha.nnel  C  when 

it  is  driven  by  a  random  sequence  of  input  letters   chosen  independently 

— > 
with  probabilities  given  by  the  vector  P     (equivalently, 'when  it  is  driven  by 


It  can  be  shown  that  for  a  memoryless  channel  any  dependence  between  input 
letters  diminishes  the  rate  of  the  channel  (see  Feinstein   [?]). 
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— > 

an  independent  source  characterized  by  P  )  is 


(3)  R(M  -  YZ    Tl  P(i,  j)  log  ^^i^i- 

bits/second.     (Recall  that  by  our  convention  one  input  letter  per  second  is 
emitted  by  the  source  and  accepted  by  the  channel.) 

To  justify  (3),  we  specialize  to  the  case  where  the  input  and  output 
alphabets  are  the  same  (so  that  in  particular  m  =  n)  and  use  Shannon's 
correction  channel  argument   [2],  which  asserts  that 

(U)  correction  rate     +  mutual   information  rate  =  source  rate   • 

The  source  rate  is  of  course  just 


m 

y  p.  log  p 

i=l       ^  ^ 


The  correction  rate  is  derived  as  follows:     Whenever  the  letter  y.  is  received, 
it  may  be  correct  or  incorrect.     Since  the  probability  that  y .  originated  from 
X.    is  p.(i)  =  p(i*j)/P'.»  1  <  ijj  <  nij  whenever  y.  is  received  we  must  supply 
an  amount  of  entropy 

m 

-XIP.(i)  logp.(i) 
i-1     ^  ^ 


to  correct  it  or  to  leave  it   stand  uncorrected  with  the  assurance  that  it  is 


More  precisely,   the  average  rate  of  correction  entropy. 
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correct.  Since  the  letter  y.  is  received  with  probability  P.,  the  average 
rate  at  which  correction  entropy  must  be  supplied  is 


m      m 

K    ^  P^(i)  log  Pj(i) 


Since 


m 


1^'^r- 


we  can  also  write  the  source  rate  as 


m   m 


m    m 


^i  Pij  ^°S  ^i  =  - 


p(i,j)  log  P. 


Finallyj  using  (U),  we  obtain 

-> 
mutual  information  rate  =  R(P  ) 


m    H  m   m  / .  .  \ 

p(i,  j)  log  p,  *T~  7Z  p(i»o)  log  ^^*^ 


s 


m    m 


p(i,j)  log  Eii*^^ 
I^  3^  P.  P' 

1  5 


which  agrees  with  (3)   for  the  case  m  =  n. 

In  what  follovra  we  shall  find  it  convenient  to  introduce  the  symbol 
<^    ^  (angular  brackets)  to  denote  averaging  with  respect  to  the  joint 
probability  distribution  p(i,j),   i.e.,   if  f(i,j)   is  a  function  of  the  two 
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integral  arguments  i  and  j,  whore  l<i<m,  l<j<nj   then 


m        n 

<f(i,3)>  =    EI    EI    P(i>5)  ^(i»3) 
1=1 


With  this  notation  (3)  becomes 


Ri?)     =-<logP(i)>     *     <logp  (i)>  =<log    £il2i)> 


p.  p; 


As  an  example  of  (3)>  consD.der  the  general  binary  channel 


(^) 


C(a,p)  = 


a     1  -  a 
P     1  -P 


,     0  <  a,     p  <  1     , 


fedbya  source  producing  O's  and  I's  independently,  with  probabilities  P    and 
P-   =  1  -  P  ,   respectively.     (Eq.    (5)  means  that  transmitted  O's  are  received  as 
O's  with  probability  a,  while   trGnsmitted  I's  are  received  as  O's  with  probability 
p.)     Then  it  is  easily  verified  that  the  rate  associated  with  (5)  and  the  input 

vector 

— > 
P      = 


xs 


(6)  R(a,p;  P  )  -aP^log  (a/P^)  +  (l-a)P^log(  (l^)/P^)  +  pP^log(p/P^  +  (l-p)P^log((l-p)/p^), 
where  P^  »  aP^  +  P(l  -  P^)  is  the  probability  of  a  received  zero  and  PJ   =  1  -  P' 


is  the  probability  of  a  received  1. 
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h,  Channel  capacity 

Following  Shannon   [2],  we  define  the  capacity  c(C)   of  the  (discrete 
memoryless)   channel  C  as  the  large?t  value  of  the  rate  which  is  achieved  when 
the  dnput  probabilities   are  varied  over  all  possible  values,   i.e. 


(7) 


Max  -> 

c(c)  =        -^       R(P) 
P 


As  an  example  consider  the  PSC  where  the  channel   matrix 


syia 


a       p 


is  obtained  by  setting  p  =  1  -  a   in  (5).     In  this  case  it  is  clear  from 

symetry  (and  it  can  be  verified  by  direct  calculation)   that  capacity  is 

achieved  for  the   choice  P     =   P^    =  1/2,   which  implies  P     =  P-    =  l/2  as  well, 

o         1  '  01 

Substituting  these  values  in  (6)  and  using  [3  =  1  -  a,  we  find 


c(C        )   =  1  +  a  log  a     +     fi  log  B 
sym  ^  M       &  ^ 


for  the  capacity  of  the  BSC. 

The  fundamental  importance  of  the  channel  capacity  as   an  information- 
theoretic  quantity  stems  from  the  role  that  it  plays  in  Shannon's  second 
coding  theorem   (+he  so-called  noisy  coding  theorem),  which  it  is  safe  to  say 
contains  most  of  the  substantive  content  and  technical  promise  of  information 
theory  (taken  together  with  corresponding  studies  of  finite-length  block  coding 
like    [1]).     This  theorem  asserts  that  with  proper  encoding  it  is  possible  to 
transmit  information  at   any  rate  less  than  capacHj'"  with  arbitrarily  snail 
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probability  of  error,  provided  th?t  the  block  length  of  the  code  is  long  enmagh, 
?.nd  furthermore  that  regardless  of  the  encoding  scheme,   errors  will   always  be  made 
if  one  gttem.pts  to  tra  nsmit  information  at  a   rate  greater  than  capacity.     Much 
space  has  been  devoted  in  the  literature  to  a  rigorous  demonstration  of  this 
theorem  for  an  appropriately  large  class  of  sources  and  channels,   and  the  whole 
subject  is   s  difficult  one  which  we  shall  not  go  into  here.     The  interested 
reader  is  referred  to  the  papers  of  Khinchin   [3]   and  the  book  by  Feinstein   [7] ; 
the  latter  author  played  an  important  role  in  developing  a  rigorous  proof  of 
the  noisy  coding  theorem. 

We  turn  now  to  the  question  of  how  the  mathematical   operation  symbolized 

by  (7)  is  to  be  carried  out  in  general.     This  operation  involves  more  than  a 

— > 
simple  maximization  problem,   since  the  vector  P     in  question  is   subject  to  the 

constraint  that  it  be  a  vector  with  non-negative  conponents  which  add  up  to 

vrnity.     We  begin  by  writing  (3)  in  a  form  which  explicitly  exhibits  its 

dependence  on  the  input  probabilities  P. j   this  form  is 

— >  ,  ^  m.       n  m  m      n 


( 8)  R( p!  -  -  <  log  P^ >.  <  log  Pij>  -  Z^  SZ^iPi jl°g^PiPi j  *  ^  ^  ¥i jl°S  Pij  • 


To  incorDorate  the  constraant 


m 


S^^  =  ^ 


we  add 


m 
1=1 


For  the  time  being  we  neglect  the  additional  constraint  that  Pj^>0   ,   l<i<m  , 
which  will  be  discussed  below  in  connection  with  Muroga's  work. 
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to   (8),  whei^  X  is  an  undetermined  Lagrange  multiplier.     Differentiating  the 
sum  of  these  two  terms  v:ith  respect  to  P. ,   1  <  i  <  m  ,   and  equating  the  result 


to  zero,  we  obtain 


(9)  T~p^    log-^ir^ =  ^i  ,         l<i<m    , 


i=l   ^  ^-^ 


where  \x  =  \  -  (l/log  e)   is  a  new  constant.     Multiplying  (9)  by  P.    and  summing 
over  i,  we  find  that 


where  c  is  the  capacity  of  the  channel  C. 

Suppose  now  that  the  channel  C  under  consideration,  with  channel  matrix 
(2),   is  squall  (m  =  n)   and  that  det(C),  the  determinant  of  (2),   is  non-vanishing. 
Then  the   inverse  matrix  C"   ,  satisfying  the  matrix  equation 

{ID)  CC"-"-  =  C'-'-C  =  I 

(where  I  denotes  the  unit  matrix)  exists.     If  we  denote  the  elef.ents  of  C 


-1 

'^6 


by  pT^  ,   1  <  i,   j  <  m  ,  then  (3D)  reads 


m  ^         ri         -. 

IZ  Pi  iPik  =  EI  Pii  Pik  '  ^k     '         1  <  i,k  <  m     , 


^   ^ij^jk      ^  ^ij  ^jk       -ik     ' 

where  6..    is  the  Kronecker  delta  symbol,  equal  to  1  when  i  =  k  and  0  when 
i  7^  k,     Miiltiplying  (?)  by  pT.    and  summing  over  i,  we  obtain 
in      m        -|  in  m         -, 


-  12  - 

Hence 

in  f      m        ,  m      n      _-, 


1=  \Pik  ■  «^(-=^p;:i  *  ^  ^  hxHi  1^^  hi}'  1  ^ "  ^ 


m 


where  by  exp(x)  we  mean  2^,   the  base  appropriate  to  our  choice  of  units. 
Multiplying  by  pT.  and  smnming  over  k,  we  obtain 

(11)  Pi  -  ^  ffi  e^  |-c^  lii  *  ^IZ  P^T^ti  l°g  Pijj  .     1  <  i  f  m  . 

Finally,   following  Muroga   [8] ,  we  note   that 

so  that  (11)  can  be  simplified  to 

(13)       Pi  -  ^  Pi^  «^  l-"^  *  ^  ^  Pk^'iJ  l°g  PiJ  j  '     l<if"'- 

Thus,   the   channel  capacity  is  the  number  c  which  when  substituted  in  (13) 
gives  Pj  >  0  and 

m 


^'-' 


Then  with  this  value  of  c,   (13)  gives  the  corresponding  rate-maximizing 
values  of  the  iiput  probabilities  P.,  l<i<m  .     Explicitly,  we  form  the 
sum 


and  use  the  relation  (12)  again,  obtaining 
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m      m 


n       m 


1  = 


m  1  1.1        111  -1 

|:  P^^  exp     -c  .  1^  1^  p-. P.  .  log  p.  . 


n 


m      n 


exp 


1 
-'•  *  g  ^  P^  Pl3  1°' Pijj      • 


n      m 


Solving  for  c,  we  get 

m 
(lU)         c  =  log      )>       exp 


where  it  wiDJ.  be  recalled  thi?.t.  both  log  and  exp  are  taken  to  the  base  2. 

Computations  are  simplified  by  defining  (after  Muroga   [8])  the  auxiliary- 


vector 


— > 

X  = 


x„ 


m 


which  satisfies  the  matrix  equation 


(15) 


->        -> 

C  X     =  -  H        , 


— > 


where  H     is  the  row-entropy  vector  of  the  channel  0,   i.e.  the  vector 


(16) 


— > 

H 


m 


m 


m 


Plilog 

Pli 

Pjsioe 

P2i 

Prrdl°S 

Pmi 

-  11; 


-1 


It  follows  from  (l?)  and  (16)  that  X  =  -C""^  H  ,  i.e.  that 


in    ^  in 
X.  =  T~'  p7  .  }       p .,  log  p .,  ,  1  <  i  <  n  , 


— > 


so  that  in  terms  of  the  components  of  X  ,  (lU)  becomes  simply 


(17) 


n 
c  =  log  )>       exp  X. 


Moreover  (13)  becomes 


in 


-1 


exp(-c)  y^  p^^  exp  Xj^  ,     ^  <  i  < 


ra 


Since 


-1 


p^l  =  cof(p,jJ/det(C)     , 


x^here   cof(p..  )  Is  the  cofactor  of  the  element  p.,  ,  we  finally  have 

Pll Plm 

exp(X-)   ....   exp(X  ) 
X  m 


(18) 


p    _  exp(-c)        .. 
1         det(C) 


^■ml' 


■mm 


where  the  matrix  in  (l8)  differs  from  the  channel   matrix  C  by  having  the 
entries  exp(X^),    ...,   exp(X   )  instead  of  p.^,    ...,   p.      in  the  i'th  row. 

The  method  just  described  requires  modification  if  it  leads  to  negative 
input  probabilities,   and  more  generally  if  det(C)  =  0  or  the  sizes  of  the  input 


-1^- 


and  output  alphabets  are  different.     Details  of  how  to  deal  with  these  various 
cases   are  given  in  MuiHDga's  paper  [8],   to  which  the  interested  reader  is 
referred.     There  is  also   available  a  different  and  more  easily  visi-'.alized 
approach  to  the  general  problem  of  capacity  due  to  Shannon   [9] . 

We  conclude  our  present  discussion  by  discussing  a  one-parameter 
family  of  ternary  channels  which  for  suitable  values  of  the  parameter 
leads  to  a  negative  probability  for  one  of  the  input  sj^bols,  which  must 
therefore  then  be  suppressed  according  to  Muroga.     Consider  the  channel 


C  = 


where  0  <  a  <  1.     The 


,-1 


a         1-a 

0 

1      ° 

1 

0         1-a 

a 

i  inverse  ma 

trix  is 

1 

25: 

^     -^ 

1 

a              1 

2(l-a) 

a-1       2Cl-a; 

1 
-  ^a 

1 

1 
2a 

The  row-entropy  vector  is 


H 


H 
1 
H 
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— > 
and  the  vector  X  is 


— > 

X  = 


-1 
H-a 

1-0, 

-1 


where 


H  =  -a  log  a  -  (1  -a)  leg  (l  -  a  ) 


Using  (17)  and  (l8)  t-tc  find  that  the  sysnbol  corresponding  to  the  middle  row  of 
C  should  be  used  i^ri. th  the  probability 


1    a     /   H-a 


1  +  exp 


if  capacity  is  to  be  acMeved.  This  symbol  should  be  suppressed  when  the 
numerator  goes  negative.  This  happens  when  a  >  a  ,  where  a  is  the  solution 
of 


H(a  )  -  a 
^  0     o 

1  -  a. 


a 


=  log 


T-a" 


or 

(19) 


log  a  =  -  a 
^  o     o 


The  solution  of  the  transcendental  equation  (19)  is  ct^^  0.6m.  When  p  >  0, 
the  capacity  of  the  channel  is 

log  <  1   +   exp  |-  -y^  j  \     bits/second  , 
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whereas  when  p  5  0,  the  channel  matrix  reduces  to 

o  1-0   0 
0  1-0   a 

This  is  the  matrljc  of  the  binary  erasure  channel  (BEC),  i.e.  the  channel  in 
which  O'a  and  I's  are  transmitted  and  O's,  I's  and  x's  are  receivedj  a  is  the 
probability  that  a  0  or  1  is  received  correctly  and  l-o  the  probability  that 
a  0  or  1  is  received  as  an  x.     Since  received  O's  and  I's  are  alwajrs  correct, 
irtiereas  a  received  x  is  always  wrong  and  equally  likely  to  have  come  from  a 
0  or  a  1,  the  capacity  of  the  BEC  is  obviously©  bits/second.     It  is  interesting 
to  note  that  the  capacity  of  C  is  1  bit/second  both  when  o  »  0  and  a  ■  1,  but 
when  0-0  the  s^rmbol  corresponding  to  the  middle  row  should  be  sent,  whereas 
when  o  -  1  it  should  be  suppressed. 

5.     The  general  binary  channel 

Using  Muroga's  method,  Silverman  [lO]  has  made  a  detailed  study  of  the 
general  binary  channel  C(a,p)  defined  by  (5)  •  We  sketch  without  proof  some 
of  the  results  obtained  in  his  paper: 

1)  The  capacity     c(a,p)  of  the  general  binary  channel  is  given  by  the 
formula 


-pH(o)  +  aH(p) 

c(a,p)  «  +  log 

p  -a 


1  +  exp 


'H(a)  -  H(p)^ 
P  -  o 


The  function  c(o,p)  has  the  synmetries 

c(a,p)  -  c(p,a)  -  c(l  -  a,  1  -p)  -  c(l  -  p,  1  -  a)  , 


loeb   [11]   has  also  studied  some  aspects  of  the  general  binary  channel. 
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and  defines  a  surface  over  the  unit  square  0  <  a  ,  p  <  1*  Lines  of  constant 
c(a,p)  are  shown  in  Fig,  1 

2)  Capacity  is  achieved  if  O's  are  transmitted  with  probability 


P«(a,p)  =  P(p-a)"^  -  (p-a)"^ 


P^(a,p)  satisfies  the  relation 


1  +  exp 


^H(p)  -  H(a)^ 
p  -o 


-1 


0.37 ^i  <  P  (a,p)  <  1  -  i  —  0.63   , 


which  explains  the  footnote  in  Section  1,  The  function  P  (a,p)  is  discontinuous 
at  the  points  a  «  p  =  0  and  a  «  p  «  1;  it  has  the  syinmetries 

P  (a,p)  =  P^(l-a,  1-p)  -  1  -  P„(P,a)  »  1  -  K(l-^,   l-«)  . 


Lines  of  constant  P  (a,p)  are  shown  in  Fig.  2. 

3)  At  capacity,  O's  are  received  with  probability 


p:(a,p) 


1  +  exp 


^H(p)  -  H(a) 
p  -o 


-1 


The  function  P'(a,p)  has  the  symnetries 

P^(a,p)  '  P^(p,a)  -  1  -  P^Cl-a,  1-p)  «  1  -  P*(l-P,  1-a)     • 

Lines  of  constant  P'(a,p)  are  shown  in  Fig.  3. 

The  reader  interested  in  other  properties  of  the  general  binary  channel, 
e.g.,  the  form  of  the  channels  giving  the  maximally  asymmetric  input  probability 


distributions  (P  '^  l/e  or  P  '-^l  -  l/e),  the  probability  of  error  for  th© 
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a 

Fig.   1  -  Lines  of  constant  c(a,p). 


1.0 
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Fig.  2  -  Lines  of  constant  P  (a,p). 
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Fig.   3  -  Lines  of  constant  P  (a,p). 
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general  binary  channel,  etc.,  is  referred  to  Silverman's  paper   [lO] . 

6.     Channels  vdth  fading 

The  coraraunication  situation  dealt  with  so  far  can  be  indicated  schema- 
tically by  the  diagram 

T  ->  C  ->  R     , 

i.e.,  information  is  sent  from  a  transmitter  T  to  a  receiver  R  through  a  channel 
C,  characterized  by  a  stochastic  matrix  (see  Section  3)«  (We  make  no  distinction 
between  the  primary  information  source  S  of  Section  2  and  the  transmitter  T, 
although  in  general  there  is  the  problem  of  (noiselessly)  encoding  the  output 
of  S  so  as  to  match  the  input  of  T.)  We  now  consider  the  following  generali- 
zation of  this  situation:  Instead  of  one  (memoryless)  channel  C,  let  there  be 
a  family  of  (memoryless)  channels  C  ,  where  the  index  a  ranges  from  1  to  s, 
and  let  the  channel  which  is  actually  present  at  a  given  time  of  transmission 
depend  on  the  state  of  a  random  device  N  (N  for  "  nature").  Schematically,  we 

have 

N 

1 

T  — ►C  — >  R  , 

i.e.,  infonnation  is  sent  from  the  transmitter  T  to  the  receiver  R  through  the 
channel  C,  the  state  of  which  depends  on  the  state  of  nature  N.     We  asstane 
that  the  state  a,  1  <  a  <  a  ,   chosen  by  nature  is  statistically  independent 
of  past  and  future  states  of  N  and  of  the  transmitted  sequences  as  well;  let 
p     be  the  probability  that  nature  chooses  the  state  o.     The  channel  C  is  now 
represented  by  the  family  of  stochastic  matrices 
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C     - 
a 


^11^°^     ^12^*^^   ••••  '^n^'^^ 


P2i(a)     P22^"^  ••••  P2n^^^ 


^tnl^"^     W°^   ••••  Pmn^*^^ 


,  1  <  a  <  s 


Since  the  transition  probabilities  p,  .(a)  are  now  random  variables,   a  new 
element  of  randomness  has  entered  into  the  problem.     Such  a  model  might  be 
used  to  give  an  abstract  representation  of  communication  in  the  presence  of 
"  fading ••  ,   and  with  this  in  mind  we  refer  to  the  totality  of  channels  C  , 
l<a<s,   asa  channel  with  fading . 

A  natural  problem  in  the  theory  of  channels  with  fading  is  that  of 
finding  the  channel   capacity  for  various  conditions  of  knowledge  of  N  at 
the  transmitter  and  receiver.     There  are  four  possible  cases,  which  can  be 
schematically  represented  as  follows: 

N 

1 

Case  1.  T    — »  C     — ♦  R 


Case  2* 


Case  3* 


Case  U» 


T    — »  C    -+  R 
N 

T    -^  C    — >  R 
N 

T    — »  C    -^  R 
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Case  1,  which  we  ha^e  already  encountered,  represents  the  situation  in  which 
neither  the  transmitter  nor  the  receiver  knows  nature's  state,   so  that  N  is 
effectively  just  more  noise  in  addition  to  that  already  included  in  C,     Case 
2  is  the  situation  in  which  the  receiver  but  not  the  transmitter  knows  nature's 
state.     Case  3  is  the  situation  in  which  both  the  transmitter  and  the   receiver 
know  nature's  state,   and  Case  k  is  the  situation  in  which  the  transmitter  but 
not  the  receiver  knows  natujre's  state.     We  now  give  expressions  for  the 
capacity  in  all  four  cases. 

Let  p(a,i, j)  be  the  joint  probability  that  nature  chooses  the  state  a,   that 
X.   is  transmitted  and  that  y.  is  received.     In  cases  1  and  2,  where  the 
transmitter  is  ignorant  of  nature's  state,  we  have 

(20)  P(a,i,j)  =  PaPiPij^''^       • 

In  case  3j  where  the  transmitter's  action  can  depend  on  nature's  state,  we  have 

(21)  p(a,i,j)  .  P^Pi,^Pij(„) 

where  P.       is  the  probability  of  choosing  x., given  that  nature's  state  is  a, 
(Case  h  requires  special  treatment^   see  below.)     Generalizing  the  angular 
bradcet  notation  of  Section  3,  we  write 

3      m      n 
<f(a,i,j)>-    XZIZII  P(<^*i»J)  ^(«>i»J)       • 

We  now  derive  expressions  for  the  channel  capacity  in  the  four  different  cases. 
Case  1.  Since  neither  the  transmitter  nor  the  receiver  knows  nature's. 
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state,  the  channel  has  the  same  capacity  as  the  average  channel 

_      3 

'  -  '^a  a 

0=1 

with  transition  probabilities 


Pi  J  "  tl  ''^^iJ^"^   • 


Explicitly  we  have,  from  (8) 


Max   /       PiJ 


c^  -  ^-  <  log 


P   \     m 


^iPij 


for  the  capacity  in  this  case. 

Case  2.  Since  now  the  receiver  can  use  its  knowledge  of  the  pair 
(ct>y^)  "to  infer  x. ,  we  have,  from  (3) 

Max    /     p(a,i,j) 
(22)  c„  -  -^  <  log 


P    \     P.p(a,j) 


for  the  capacity  in  this  case,  where  p(a,j)  is  the  probability  of  the  pair 
(a>yj)»  i.e. 

m 


P(a,j)  -  ZI    P(«»i»J 


Using  (20)  we  can  rewrite  (22)  as 
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Max 


(23) 


log 


Pi3^«> 


m 


PiPij(a) 


Let  R  (P)  be  the  rate  in  the  channel  C  when  using  the  input  vector  P  »  Then 
a  a 

(23)  is  just 


(2U) 


Max    s 

— > 
P   a 


Pa^a(P) 


In  other  words,  the  capacity  in  case  2  is  the  maximum  average  rate  when 
driving  all  the  channels  C    with  the  same  input  probability  vector  P.     Clearly, 
c-  >  c^   ,  since  we  are  now  using  more  detailed  knowledge  of  the  channel. 

Case  3.     Now  the  transmitter  can  base  its  choice  of  input  probabilities 
on  nature's  state,  choosing  a  suitable  input  vector 


1,* 

^2.a 


m,a 

for  each  state,  1  <  o  <  s.  Moreover,  as  in  case  2,  the  receiver  has  the  pair 
(o,y-)  available  to  infer  x. .  Therefore  we  have 


(25) 


Max 


log 


p(a,i,j) 
Pl,aP(a,j) 
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— > 


for  the  capacity  in  this  case,  where  the  maximization  is  over  all  P     , 
1  <  a  <  s  .     Using  (21),  we  can  rewrite  {2$)  as 


Max        s  — > 

a 


In  other  woivis,  the  capacity  in  case  3  is  the  maxbiura  average  rate  when 

— > 
driving  each  channel  C     with  its  own  inptt  probability  vector  P     .     It  follows 

at  once  that 

(26)  c     -ZIPa<^(C   )   , 


i.e.,  the  capacity  is  just  the  average  capacity  of  the  channels  C     , 
1  <  a  <  3.     Clearly  c.  >  Cp  ,  since  now  the  transmitter  as  well  as  the 
receiver  is  using  the  extra  information  about  the  channel. 

Case  h  is  more  complicated.     In  this  case  Shannon  [l2]  has  shown  that 

the  capacity  of  the  channel  with  fading  is  the  same  as  the  capacity  cf  a  new 

m     X    n     (non-fading)  channel  C  ,  with  input  letters  consisting  of  the  m 

s-tuples  (x.    ,  X.    ,   ...,  X.    ),  1<  i-,  i«,   .,.,  i    <  m    and  output  letters 

12  s 

consisting  of  the  n  letters  y.,  1  ^  J  5  n>  where  the  transition  probabilities 

for  C     are  defined  by 


Pi,.!,,  ....  1,   '   ^-gVy 


(«) 


Thus  the  capacity  in  this  case  is 

c,  -  c(c')  , 
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and  clearly  <x,  <  c.    <  c«  .    We  now  give  two  examples  which  illustrate  the 


yanous  cases. 


Exainple  1.     Suppose  nature  has  two  states  1  and  2  with  probabilities 
a  and  p,   respectively,   and  suppose 


1    0 
0     1 


0  1 

1  0 


Then  the  average  channel  C  is  the  BSC,  i.e. 


-  oC^  +  pCg  - 


so  that  c-  -  1  +  o  log  o     +    p  log  p  bits /second  (see  Section  U).     If  nature's 
state  is  known  by  the  receiver  or  by  both  the  transmitter  and  the  receiver, 
then  the  capacity  of  the  fading  channel  is  obviously  1  bit/second,  achieved 
by  transmitting  O's  and  I's  with  equal  probability  aiKl  interchanging  O's  and 
I's  at  the  receiver  when  the  channel  is  in  state  2.     Eqs.    (2h)   and  (26)  confirm 
that  Cp  =»  c_  =  1.     When  only  the  transmitter  knows  nature's  state,  the 
capacity  is  again  1  bit/second,   achieved  by  transmitting  O's  and  I's  with 
eqiial  probability  and  interchanging  O's  and  I's  at  the  transmitter  when  the 
channel  is  in  state  2.     In  this  instance  (case  U)  Shannon's  construction  asserts 
that  Ci    =  c(c'),  where 


a  p 

1  0 
0  1 

P  a 
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-^     ->     -> 

Denoting  the  rows  of  this  matrix  by  the  vectors  A^  ,.,,  A,  (e.g.  A^^ 

we  have 

— >   ->   -o 

A^  =  aAg  +  pA 

Aj^  -  pAg  +  oA^ 


(a,p)  ), 


It  follcws  by  an  argument  due  to  Shannon  [9]  that  the  symbols  corresponding 
to  the  first  and  fourth  rows  of  C  have  to  be  suppressed  if  capacity  is  to  be 
acMeved,  Dropping  the  first  and  fourth  rows  of  C  ,  we  get  the  matrix  C^  , 
so  that  c,  «  c(C,)  =  1  bit/second,  as  required. 

Example  2.  Let  there  be  three  states  with  equal  probability,  and  let 


^1- 


1     0 

1  1 

2  1 


^2- 


1  1 

2  2 
1  1 
1  1 


1  1 

2  2 

0     1 


Then  if  neither  the  transmitter  nor  the  receiver  knows  nature's  state,  the 
capacity  of  the  fading  channel  is  that  of  the  average  channel 


2  1 

3  3 


1     2 
5    ^ 


i.e.  c^  -  c(7)  .  1  ♦  (1/3)  log  (1/3)  +  (2/3)  log  (2/3)  -  (5/3)  -  log  3  -^.082 
bits/second.     If  only  the  receiver  knows  nature's  state,  then  by  (2U) 


Max 

T  3 


i      [r^(P^  ♦  R^iT)  +  H^{?)] 
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By  symmetry,   the  maxirauBi  is  achieved  for 


— > 
P 


and  an  elementary  calculation  shows  that  c^  ■  1  -  (l/2)  log  3   '^     '^OS 
bits/second    >    c,    .     If  both  the  transmitter  and  the  receiver  know  nature's 
state,  then  by  (26) 


I  =(=i' 


since  c(C,)  -  c(C   )  and  c(C_)  =0.     Applying  Muroga's  method  (see  Section  U), 
we  easily  find  that  c(C-)  ■  log  5  -  2  ,   so  that  c^  "  o  1°S  5  -  ■,  -^  .215 
bits/second    >     c-     , 

If  the  transmitter  but  not  the  receiver  knows  nature's  state,  then 
according  to  Shannon's  construction,  c.    =  c(C   ),  where 


C' 


2  1 

3  3 

1  1 

2  2 

2  1 

3  3 

1  1 

?  2 

1  1 

1  2 

3  3 

1  1 

?  1 

1  2 

3  3 


Driving  any  binary  channel  with  a  source  emitting  O's  and  I's  with  equal 
probability  leads  to  a  rate  very  near  capacity,  which  explains  why  c_  is 
so  near  Cp  (see  footnote  to  Section  l). 
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By  a  simple  extension  of  the  argument  given  in  connection  with  the  preceding 
example,  we  have 


c     -  c(C')  "  c(C)  -  c,  '^  .082  bits/second  , 


so  that  in  this  case,  tmlike  example  1,  the  transmitter  cannot  use  its 
knowledge  of  nature's  state  to  increase  the  channel  capacity. 

7.  Cascaded  channels 

Let  C  be  an  /  X  ni  channel  matrix  with  output  alphabet  y^,  ...,  y  ,  and 
let  C  be  an  m  ><  n  channel  matrix  with  input  alphabet  y,,  ...,  y  (i.e. 
identical  to  the  output  alphabet  of  C).  Then  we  agree  to  apply  y^  to  the 
input  of  C  whenever  y^  is  received  at  the  output  of  Cj  this  mode  of  channel 
combination  is  called  cascading.  Denote  by  p.  and  p.  .  the  elements  of  the 

I  It  M 

matrices  C  and  C  ,  and  denote  by  p.  .  the  elements  of  the  matrix  C   obtained 
by  cascading  C  and  C  ,  as  just  described.  Then,  since  obviously 


m 


S'^i 


C       is  obtained  from  C  and  C     by  matrix  multiplication,  i.e.  C       =  CC   .     Since 
matrix  multiplication  is  in  general  non-comnutative,  the  same  is  tnie  of 
channel  cascading. 


The  problem  of  cascading  identical  binary  channels  C(a,p) 
especially  simple,   in  view  of  the  identity 


a  l-a 
p  1-p 


is 


(27)       cV,p)     s 


p(n)  ^_p(n) 


1-a+p 


p  1-a 
p  1-a 


-  (a-p)° 


o-l    1-a 
p       -p 
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which  is  easily  derived  from  a  general  fonnula  of  matrix  algebra  (see  e.g.   [l3]). 
(We  exclude  the  trivial  cases  a=l,  pO  and  a«0,  p-1,   corresponding  to  the 


noiseless  channels 
be  derived  from  (27): 


1  0 
0  1 


and 


0  1 

1  0 


.)     A  number  of  simple  consequences  can 


l)  Since   |a  -  p|  <  1,  we  have 


(28)  ^^^  C%,p)  =  C^°°^c,p)  -  — 


1-a+p 


p  1-a 
p  1-a 


^(00)   ^^(oo) 
p(oo)  ^_p(oo) 


The  limiting  channel  C    (a,p)  obviously  has  zero  capacity,  in  accord  with 
the  intuitive  idea  that  an  infinite  cascade  of  noisy  channels  must  destroy 
any  information  fed  into  it. 


2)  Since 


l-o 


|(oo)    p(n)_  p(oo) 


a-1 


«11  the  channels  C^(a,p)  lie  on  a  straight  line  in  the  (a,p)  square  passing 
through  the  point  corresponding  to  the  limiting  channel  C    (a,p)  and  the 
point  (0,1).  Thus,  referring  to  Fig.  1,  we  have  the  following  simple  interpre- 
tation of  the  operation  of  cascading  a  binary  channel  with  itself:  Using  (28), 

draw  the  straight  line  just  described;  then  as  n  increases,  C  (a,p)  approaches 

(en) 
C    (a,p)  along  this  straight  line,  moving  alternately  from  one  side  of  the 

zero-capacity  line  a  ■  p  to  the  other  if  a  -  p  <  0  . 

The  operation  of  channel  cascading,  or  equivalently  of  multiplying 
channel  matrices,  is  a  partial  ordering  in  the  following  sense:  Given  any  two 
channels  C.  and  C^,  then  C-  is  said  to  include  Cg,  written  C^  ^  '^2  *  ^  there 
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exists  a  channel  C  such  that  C^C  «  C-  ,  i.e.  if  C^  can  be  obtained  from 

C^  by  cascading.  Channel  inclusion  has  the  defining  properties  of  a  pjartial 

ordering,  namelj 

1)  C  ^  C  ,  for  any  C  ; 

2)  C,  ^  Cg  ,  Cp  ^  C-  implies  C^  =  C.  . 

On  the  other  hand,  given  arbitrary  channels  C^  and  C^  ,  neither  of  the 
relations  C^  ^  C„  ,  Cp  ^  c,  may  hold,  i.e.  C,  and  C^  may  not  be  comparable  . 

In  the  case  of  binary  channels,  the  structure  of  the  partial  ordering 
is  very  simply  displayed.  Suppose  we  have  two  binary  channels 


a  1-a 
P  1-P 


C' 


a'  1-c' 
P'  1-P' 


and  assume,  as  we  can  without  loss  of  generality,  that  p  <  a,  p  <  <*  •  Then, 
following  a  procedure  suggested  by  Bimbaum  (private  communication),  we  first 
prove  the  following  lemma: 

A  necessary  and  sufficient  condition  for  C  ^  C  is  that  the  interval 
(L-,L«)  contain  the  interval  (L.,  !«),  where 

I^  -  p/a  ,  I^  -  p'/a'  ,  Lg  -  (l-p)/(l^)  ,  L^  -  (l-p*  )/(l-a') 

(Note  that  in  ovir  case  0<'L^<l<L2<(x>,     0<iJ<1<l'<oo.) 

lb  show  the  necessity,  we  suppose  that 
X  l-x 

y  1-y 


If  c(C)  denotes  the  capacity  of  C,  then  c(C)  >  c(c')  is  a  necessary  but  not 
sufficient  condition  for  C  :3  C'.  *" 
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is  the  channel  such  that  CC"  «  C   .     Then,  doing  the  niatrix  multiplication 
explicitly,  we  find  that 

^  ox+(l-o)y    ""  ^ 

since  y  >  0,  and 

l!  -  (l-p')/(l-a')  .  l-p^-(l-f)r  <  (i-p)/(i^)  -  r      , 
^  l-ax-(l-c)y~  ^ 

since  z  <  1. 

To  show  the  sufficiency,  we  solve  formally  for  the  parameters  x,y  of 

C",  finding 

oB'  -c'b 

7  "  -^ '^      * 

a  -p 

g'-p'      . 

Since  a-p     >0,  a-p>0,  and  ap'  -  o'p  >  0  by  the  hypothesis  that  L,  <  I*,    , 
we  see  that  0  <  y  <  x.     Moreover  7  <  x  <  1,   since  ap'-  a'p  +o'-p    <a  -p 
is  an  easy  consequence  of  the  hypothesis  Lp  <  L^.     This  completes  the  pr«of 
that  C  2  C     and  (L,!.^)  2  (^>^)  ^'^  equivalent  statements. 

Using  the  lemma,  we  can  give  a  simple  geometrical  model  of  the  partial 

ordering  of  binary  channels  under  cascading.     Plot  the  point   (a,p)  cojrresponding 

to  C  in  the  unit  square,  and  draw  the  strai^t  lines  from  the  point  (a,p) 

to  the  points  (0,0)  and  (1,1),  as  shown  in  Fig,  U,     Then  sill  the  points  in 

the  region  shaded  with  vertical  lines  represent  channels  contained  in  C, 

i.e.  channels  which  can  be  obtained  from  C  by  cascading.     This  follows  by 

applying  the  lemma  in  the  part  of  the  region  lying  below  the  line  a-p,   and 

0  1 


then  noting  that  multiplication  by 


1  0 


changes  a  channel  into  its  reflection 


in  the  line  a-p.  Moreover,  it  follows  from  this  constiruction  that  all  points 
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in  the  region  shaded  by  horizontal  lines  represent  channels  containing  C. 
Similarly,  points  lying  outside  of  the  shaded  regions  represent  channels 
which  are  not  comparable  with  C,   i.e.  which  cannot  be  reached  from  C  either 
by  preraultiplication  or  postraultiplication  by  any  binary  channel. 


Fig.  h     -     Illustrating  the  partial  ordering  of  binary  channels. 

For  further  discussion  of  topics  related  to  channel  cascading,  see 
Shannon  [1I4] ,  Bimbaum   [lS\   and  Silverman   [1D]  . 

8.     Channels  with  memoiy 

We  now  consider  a  simple  idealized  oominuni cation  system  suggested  by 
Chang  [16],  which  leads  naturally  to  the  study  of  channels  with  memory. 
Let  the  channel  input  be  a  source  emitting  binary  digits,   and  let  the  channel 
be  such  that  O's  and  I's  emitted  at  the  time  t  -  0   are  represented  by  the 
waveforms 

f^(t)     -     A  exp  (-t/'C)   , 
f^(t)  -  -A  exp  (-t/r  )   , 
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respectively,  in  the  absence  of  noise  .  The  effect  of  noise  will  be  repre- 
sented  by  the  addition  of  white  Gaussian  noise   of  r.m.s.  voltage  <y  . 
Suppose,  to  consider  the  simplest  non-trivial  case,  that  the  source  produces 
the  digits  in  isolated  doublets,  i.e.,  let  a  pair  of  binary  digits  separated 
by  an  amount  y  <'^    be  transmitted,  and  then  let  there  be  a  pause  in 
transmission  long  enough  to  allow  substantially  complete  decay  of  the 
exponential  expC-t/f).  Finally,  suppose  that  we  use  synchronous,  threshold 
detection  at  the  receiver,  i.e.,  at  the  times  t  and  t  +  y  corresponding  to 
transmission  of  a  doublet,  we  interpret  a  positive  voltage  as  a  0  and  a 
negative  voltage  as  a  1, 

We  now  proceed  to  find  the  capacity  of  this  simple  channel.  At  a  time 
t  such  that  the  channel  has  recovered  from  the  effects  of  previous  transmitted 
signals  the  channel  is  described  by  the  matrix 

a  1-a 
p  1-p 

where 

a  -  Prob  [f^(0)  +  5  >  O]  , 

p  -  Prob  [f^(0)  ♦  C  >  O]  , 

2 

and  5  is  a  Gaussian  random  variable  with  mean  0  and  variance  (f    .  (Specifically, 


We  have  in  mind,  for  example,  a  situation  where  O's  and  I's  are  encoded  into 
sharp  positive  and  negative  pulses  of  amplitude  A  and  then  sent  through  a 
channel  whose  transmission  characteristics  resemble  those  of  an  RC  filter 
with  time  constant  f  . 

By  "  white"  noise,  we  mean  noise  with  a  constant  power  spectral  density,  or 
equivalently  with  a  correlation  function  which  is  a  delta  function.  Gaussian 
white  noise  has  no  memory.  This  is,  of  course,  a  limiting  case,  approached 
only  when  the  noise  band  width  is  much  greater  than  that  of  the  signals. 
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a  is  the  probability  that  a  0  transmitted  at  time  t     is  interpreted  as  a  0, 
and  p  is  the  probability  that  a  1  transmitted  at  time  t     is  interpreted  as  a  0, 
Similarly,  a   ,  p     and  a^ ,  p^   are  the  corresponding  probabilities  at  time 
t     +  Y»  under  the  hgrpothesis  that  a  0  or  a  1  was  transmitted  at  time  t  , 
respectively  (see  below)   ).     Clearly  we  have 


a  -  Prob  (^  >  -A)  -  1  -  F(-A)  , 
p  -  Prob  (C  >  A)  -  1  -  F(A)   , 


where 


F(x)  - 


V5n  & 


-00 


2/2<^2 


du 


is  the  distribution  function  of  the  random  variable  4»     It  follows  from  the 
synmetry  of  F(x)  that  a  +  p  =  1,  i.e.  that  C  is  a  symmetric  channel.     At  the 
time  t    *  X  >  there  are  two  possibilities.     If  a  0  was  transmitted  at  tine 
t  ,  then  the  channel  is  described  by  the  matrix 


^o- 


o  o 


B       1-8 
^o       ^^o 


where 


o^  -  Prob  [f^(0)  ♦  f^(Y)  ♦  «  >  O]  -  1  -  f[-A(1+p)]    , 
pQ  •  ^^^    ^^0(0)  ♦  fi(Y)  *  K>0]   -  1  -  f[-A(1-p)]    , 


and  p  -  exp(<-r/t?).     Clearly  we  have  a     +  p     >  I  and  C     is  asymnetric, 


Similarly,  if  a  1  was  transmitted  at  t^,  then  at  time  t^  •••  Y  the  channel  is 
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described  by  the  matrix 


where 


a^  -  Prob  [f^(0)  +  f^(Y)  *  K>0]  -  1  -  f[a(1-p)]   , 
p^  -  Prob   [f^(0)  ♦  f^(Y)  +  C  >  O]   -  1  -  FCAd+p)]    . 

This  time  a,  +  p,  <  1  and  again  C,   is  asymmetric. 

To  find  the  capacity  of  the  channel  with  memory,   it  is  simplest  to 
proceed  on  a  doublet  basis  and  enlarge  the  channel  to  a  new  U  x  U  memoryless 
channel  C  ,  with  inputs  and  outputs  consisting  of  the  four  pairs  00,  01,  30 
and  11,     Thus,  we  consider  the  U  x  U  channel  matrix 


aC^     (l^)C^ 


pc^     (l-p)Cj^ 


whose  elements  are  themselves  2x2    matrices.     Following  Chang  [l6],  we 

find  the  capacity  of  C     by  using  Muroga's  method  (see  Section  U).     The 

->  , 

row-entropy  vector  H'  of  C    is  easily  seen  to  be 


H' 


H(a)  ♦  H(aQ) 
H(a)  ♦  H(Pq) 
H(p)  +  H(a^) 
H(p)  +  H(p^) 


H(a)  D    ♦  H^j 
H(p)  V    *  B^ 


where  U  denotes  the  vector 
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— >    -o 

,   and  H  ,  K.   are  the  two-dimensional 


row-entropy  vectors  of  the  channels  C   ,  C^,   respectively.     Now  we  must  find 

-> 
the  airxiliary  vector  X'   satisfying 


Since  C     can  be  written  as 


we  have 


and 


C       0 
o 

0      c. 


.'-1 


a     1-a 
p     1-p 


X' 


a 

1 

1-p 

-P 

a-l 
a 

0        c 

-1 
1 

1 

1-p    a-1 

Oo"° 

-c»      — > 

H(a)  U    +  H^ 

a-p 

-P       a 

0      c^-"- 

->      — > 

H(p)  U     ♦  E^ 

o- 

P 

(1-P) 

H 

.H(a) 
H(a) 

— >    — > 
->    -> 

->    -> 
+   (o-l)  C^-^(H(p)U  +  H^) 

♦  a  C"-^(H(p)U    +  H^) 

-1  "^      -1         "^ 

Since  C       U-C-U-U,we  have 
o  ± 


(29)      i'--J^ 


— >    — >  — >    -> 

(l-p)(H(a)U  -  Xq)  +  (a-l)(H(p)U  -  X^) 

->    ->  — >    -> 

-p   (H(a)U  -  Xq)  ♦  a(H(p)U  -  X^^) 


1 
a-p 


1-p    a-l 
-P      a 


— > 

— > 

H(o)U  - 

^0 

— > 

— > 

H(p)U  - 

h 

-W  ' 


vrtiere  we  have  introduced  the  two-dimensional  auxiliary  vectors  X  ,    X, 
corresponding  to  the  channels  C   ,  C-,   respectively,  i.e.,   satisfying  the 


— > 


,-1 


— > 


equations  X^"  -C^     H 
^  o         o       0 


— > 

and  X- 


-C 


-1 


^    -  — ^     H, ,     Then,  if  X  is  the  auxiliary  vector 
corresponding  to  the  channel  C,  i.e.  satisfying  the  equation 


— > 
X     s 

^0 

^ 

-    -c 


-1 


H(a) 
H(p) 


it  follows  from  (29)  that 


(30) 


X'   - 


^ 
^ 


— > 

1 

1-p     a-1 

^0 

a^ 

-p       a 

-> 

h 

Eq.   (30)  ejqpresses  the  four-dimensional  avixiliaiy  vector  X    of  the  expanded 

I  ->    -^ 

U  X  U  channel  C  in  terms  of  the  two-dimensional  auxiliary  vectors  X,  X 

->  I 

and  X.     of  the  binary  channels  C,   C     and  C,.     Finally,   the  capacity  of  C     in 

bits/doublet  is  obtained  from  (30)  by  using  Eq.   (17).     Ilius,   the  calculation  of 

-> 

X     and  c(C   )  is  a  simple  matter,   provided  one  has  available  a  table  of  the 

-> 
values  of  the  auxiliary  vector  X    corresponding  to  the  general  binary  channel. 

Such  a  table  is  given  in  Chang's  paper  [l6] ,  to  which  the  reader  is  referred 

for  further  details. 


The  simple  example  Just  given,  where  isolated  doublets  are  transmitted, 
illustrates  the  general  approach  to  the  problem  of  determining  the  capacity  of 
channels  with  memory  of  the  type  under  consideration.     More  generally,  one  can 
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consider  the  case  of  isolated  groups  of  n  equally  spaced  digits,  and  then  the 
case  of  a  channel  driven  at  a  uniform  rate  with  no  pauses  for  recovery  from 
the  effects  of  previously  transmitted  signals.      (Cf  course,  the  latter  case 
corresponds  to  passing  to  the  limit  n  -^oo  in  the  case  of  transmitting 
isolated  groups  of  n  equally  spaced  digits.)     One  can  also  consider  m  x  m 
channels  with  memory,  where  m  >  2.     Such  problems  have  been  studied  by  Chang 
and  co-woricers    [l6] ,    [l?]  •     Finally,   it  is  natural  to  study  the  case  where 
the  noise  itself  has  appreciable  memory  or  even  the  case  where  there  is 
statistical  dependence  between  the  noise  and  the  signals.     It  appears  that 
much  remains  to  be  done  €j.ong  these  lines. 
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Wright-'atterson  Air  Force  Base,   Ohio       (^ 
Attni     Mr.    Paul  Springer.  WCLRE-5 

Commander 

Air  Technical  Intelligence  Center 
Wright-Patterson  Air  Force  Base,   Ohio 
Attn:     AFCIN-ljBia 

Commander 

Rome  Air  Development  Center 
Grifflss  Air  Force  Base,   New  Tork 
Attn:     RCSSTL-1 

Commander 

Rone  Air  Development  Center 
Grifflss  Air  Force  Base,  New  York 
Attn:  Hr.  Donald  Dakan,  nCDE 


hio  flO)Armed  Services  Technical  Information 
Agency 
Arlington  Hall  Station 
Arlington  12,  Virginia 


Commander 

Rome  Air  Development  Center  (ARDC) 
Grifflss  Air  Force  Base,  New  York 
Attn:  Dr.  John  S.  Burgess,  RCE 

Commander 

Air  Force  Missile  Development  Center 
Rolloman  Air  Force  Base,  New  Mexico 
Attn:  HDOIL,  Technical  Library 

Director 

U.  S,  Army  Ordnance 

Ballistic  Research  Laboratories 

Aberdeen  FVovlng  Ground,   Marvland 

Attn:     Ballistic  Measurements  Laboratory 

Ballistic  Research  Laboratories 
Aberdeen  Proving  Ground,   Marylarel 
Attn:     Technical  Information  Branch 


(2) 


(2) 


) Library 

Boulder  Laboratories 
National  Bureau  of  Standards 
Boulder,  Colorado 

National  Bureau  of  Standai^s 
Department  of  Commerce 
Washington  25,  D.  C. 
Attn:  Hr.  A.  G.  HcNlsh 

National  Bureau  of  Standards 

Department  of  Commerce 

Washington  25,  D.  C. 

Attn:  Oustave  Shapiro,  Chief 

Engineering  Electronics  Section 
Electricity  and  Electronics  Div. 

Office  of  Technical  Services 
Department  of  Commerce 
Washington  25,  D.  C. 
Attn:  Technical  Renorts  Section 

(Hnclassified  only) 

Director 

National  Security  Agency 
Washington  25,  D.  C. 
Attn:  R/D  (331) 

Ho.  Air  Force  Cambridge  Research  Center 

Laurence  G.  Hanscom  Field 

Bedford,  Mass. 

Attn;     CROTLR-2  -   P.   Condon 


(5)Hq.  Air  Force  Cambridge  Resparch  Center 
Laurence  G.   Hanscom  Field 
Bedford,    Mass. 
Attn:     CROTLS  -  J.   Armstrong 


Director,   Avionics  Division  (AV) 
Bureau  of  Aeronautics 
Department  of  the  Navy 
Washington  25,   D.  C. 

Chief,    Bureau  of  Ships 
Department  of  the  Navy 
Washington  25,   D.   C. 
Attn:     Mr.  E.   Johnston,   Code  ft33E 

Commander 

n.  3.   Naval  Air  Missile  Test  Center 

Point  Mugu,   California 

Attn:     Code  366 

U.  S.  Naval  Ordnance  Laboratory 
White  Oak 

Silver  Spring  n,  Marvland 
Attn:  The  Library 

Commander 

U.  S.  Naval  Ordnance  Test  Station 

China  Lake,  California 

Attn:  Code  753 

Librarian 

U.  S.  Naval  Postgraluate  School 

Monterey,  California 

Air  Force  Development  Field  Representative 
Naval  Research  Laboratory 
Washington  25,  D.  C. 
Attn:  Code  1072 

Director 

U.  S.  Naval  Research  Laboratory 

Washington  25,  D.  C. 

Attn:   Code  2C27 

Dr.  J.  I.  Bohnert,  Code  5210 

U.  S.  Naval  Rese&rch  Laboratory 

Washington  25,  D.  C.  (Unclassified  only} 

Classified  to  i>e  sent  to: 

Director 

n.  S.  Naval  Hesearch  Lahorai.ory 

Attn:  Code  5200 

Washington  25,  D.  C. 

Commanding  Officer  and  Director 

U.  S.  Navy  Underwater  Sou:id  Laboratory 

Fort  Trumbull,  New  London,  Connecticut 

Chief  of  Naval  Research 
Department  of  the  Mavy 
Vashington  25,  D.  C. 
Attn:  Code  1:27 

Commanding  Officer  and  Director 

D.  S.  Navy  Electronics  Laboratory  (Library) 

San  Diego  52,  California 

Chief,  Bureau  of  Ordnance 
Pepartrrent  of  the  Navy 
Washington  25,  D.  C. 
Attn:  Code  Ad3 

Chief,  Bureau  of  Ordnance 
Department  of  the  Navy 
Surface  Guided  Missile  Branch 
Washington  25,  D.  C. 
Attn:  Code  ReSl-e 

Chief,  Bureau  of  Ordnance 
Department  of  the  Navy 
Washington  25,  D.  C. 
Attn:  Fire  Control  Branch  (ReSlj) 

De[;artment  of  the  Navy 

Bureau  of  Aeronautics 

Technical  Data  Division,  Code  lil06 

Washington  25,  D.  C. 

Chief,  Bureau  of  Ships 
Department  of  the  Navy 
Washington  25,  D.  C. 
Attn:  Code  8178 


DL  -  2 


Commanding  Officer 

n.  S.  Naval  Air  Development  Center 

Johnsvllle,    Pennsylvania 

Attn:  NADC  Library 

Comraareler 

U.  S.  Naval  Air  Test  Center 
Patvixent  River,   Maryland 
Attn:     ET-31?,    Antenna  Branch 

Director 

Naval  Ordnance  Laboratory 

Corona,   California 

Commanding  Officer 

U.  S.  Naval  Ordnance  Laboratory 

Corona,   California 

Attn:     Mr.  W,  Horensteln,   Division  72 

Airborne  Instruments  Laboratory,    Inc. 

1(0  Old  Country  Road 

Hineola,  New  York 

Attn:  Dr.  E.  G.  Fubini,  Director 

Research  and  Engineering  Division 

Aircom,  Inc. 
3?U  Haln  Street 
Winthrop,  Mass. 

American  Machine  and  Foundry  Company 

Electronics  Division 

10fl5  Commonwealth  Avenue 

Boston  15,   Mass. 

Attn;  Mrs.  Rita  Moravcsik,  Librarian 

Andrew  Alford,  Consulting  Engineers 
299  Atlantic  Avenue 
Boston  10,  Mass. 

Avion  Division 
ACF  Industries,  Inc. 
ROO  No.  Pitt  Street 
Alexandria,  Virginia 
Attn:  Library 

Battelle  Manorial  Institute 

50?  King  Avenue 

Attn:  Wayne  E.  Rife,  Project  Leader 

Electrical  Engineering  Division 

Columbua  1,  Ohio 

Bell  Aircraft  Corporation 

Post  Office  Box  One 

Buffalo  5,  New  York 

Attn:  Eunice  P.  Hazelton,  Librarian 

Bell  Telephone  Laboratories,  Inc. 

Whlppany  Laboratory 

Whippany,  New  Jersey 

■Attn:     Technical  Information  Library 

Pacific  Division 

Bendlx  Aviation  Corporation 

llfOO  Sherman  Hay 

North  Hollywood,  California 

Engineering  Library 

Attn:  Peggie  Robinson,  Librarian 

Bendlx  Radio  Division 

Bendix  Aviation  Corp. 

E.  Joppa  Road 

Towson  h,   Maryland 

Attn:  Dr.  D.  M.  Allison,  Jr. 

Director  Engineering  and  Research 

Boeing  Airplane  Company 

Pilotless  Aircraft  Division 

P.O.  Box  3707 

Seattle  2b,  Washington 

Attn:     R.R.  Barber,   Library  Supervisor 

Boeing  Airplane  Company 

Wichita  Division  Engineering  Library 

Wichita  1,  Kansas 

Attn:  Kenneth  C.  Knight,  Librarian 

Boeing  Airplane  Company 

Seattle  Division 

Seattle  lli,  Washington 

Attn:  E.T.  Allen,  Library  Supervisor 

BJorksten  Research  Labs,  Inc. 

P.  0.  Box  265 

Madison,  Wisconsin 

Attn:     Mrs.  Fern  B.  Korsgard 


Convalr,   A  Division  of  General  Dynamics 

Corp. 
Fort  Worth,   Texas 

Attn:     K.O.  Brown,   Division  Research 
Librarian 

Convalr,   A  Division  of  General  Dynamics 

Corp. 
San  Diego  12,   California 
Attn:     Mrs.  Dora  B.  Burke, 

Engineering  Librarian 

Cornell  Aeronautical  Laboratory,    Inc. 
hh^S  Genesee  Street 
Buffalo  21,   New  York 
Attn:     Librarian 

Dalmo  Victor  Company 
A  Division  of  Textron,    Inc. 
1515  Industrial  Way 
Belmont,   California 
Attn:     Mary  Ellen  Addems, 
Technical  Librarian 

Dome  and  Margolin,    Inc. 
29  New  York  Avenue 
Westbury,    Long   Island,   N,  Y. 

Douglas  Aircraft  Company,    Inc. 

P.O.   Box  200 

Long  Beach  1,  California 

Attn:  Engineering  Library  (C-2S0) 

Douglas  Aircraft  Co.,  Inc. 
f27  Lapham  Street 
El  Segundo,  California 
Attn:  Engineering  Library 

Douglas  Aircraft  Company,  Inc. 
3000  Ocean  Park  Boulevard 
Santa  Monica,  California 
Attn:   P.T.  Cline 

Eq,  Sec.  Reference  Files, 

Eq.  Eng.  A250 

Douglas  Aircraft  Company,    Inc. 
2000  North  Memorial  Drive 
Tulsa,   Oklahoma 
Attn:     Engineering  Library,   D-2S0 

Electronics  Communication,  Inc. 
1830  York  Road 
Timonlum,  Maryland 

Emerson  and  Cuming,  Inc. 
869  Washington  Street 
Canton,  Mass. 
Attn:  Mr.  W.  Cuming 

Emerson  Electric  Kfg.  Co. 

8100  West  Florissant  Avenue 

St.  Louis  21,  Missouri 

Attn;  Mr.  E.R.  Breslin,  Librarian 

Sylvanla  Elec.  Prod.  Inc. 
Electronic  Defense  Laboratory 
P.O.  Box  205  -  (Unci) 
Mountain  View,  California 
Attn  I  Library 

Falrchild  Aircraft  Division 
Fairchild  Eng.  and  Airplane  Corp. 
Hagerstown,  Maryland 
Attn:  Library 

Famsworth  Electronics  Company 
3700  East  Pontlac  Street 
Fort  Wayne  1,    Indiana 
Attn;     Technical  Library 

Federal  Telecommunication  Labs. 
<00  Washington  Avenue 
Nutley  10,  New  Jersey 
Attn:  Technical  Library 

The  Gabriel  Electronici 
Division  of  the  Gabriel  Company 
135  Crescent  Road 
Needham  Heights  9U,  Mass. 
Attn:  Mr.  Steven  Galagan 


General  Electric  Advanced  Electronics  Center 
Cornell  University 
Ithaca,  New  York 
Attn:   J.  B.  Travis 

General  Electric  Company 

Electronics  Park 

Syracuse,  New  York 

Attn:  Documents  Library,  B.  Fletcher 

Building  3-lb3A 

General  Precision  Laboratory,  Inc. 

63  Bedford  Road 

Pleasantvllle,  New  York 

Attn:  Mrs,  Mary  G.  Herbst,  Librarian 

Goodvear  Aircraft  Corp. 

1210  Masslllon  Road 

Akron  15,  Ohio 

Attn:  Ubrary  D/120  Plant  A 

Granger  Associates 

Electronic  Systems 

966  Commercial  Street 

Palo  Alto,  California 

Attn!   John  V.  N,  Granger,  President 

Grumman  Aircraft  Engineering  Corporation 
Bethpage,  Long  Island,  N,  Y. 
Attn:  Mrs.  A.  M.  Gray,  Librarian 

Engineering  Library,  Plant  No,  5 

The  Hallicrafters  Company 

UiOl  West  5th  Avenue 

Chicago  2b,  Illinois 

Attn:  La Verne  LaGioia,  Librarian 

Hoffman  Laboratories,  Inc. 
3761  South  Hill  Street 
Los  Angeles  7,  California 
Attn:  Engineering  Library 

Hughes  Aircraft  Company 
Antenna  Department 
Microwave  Laboratory 
Building  12,  Room  2617 
Culver  City,  California 
Attn:  M.  D.  Adcock 

Hughes  Aircraft  Company 
Florence  and  Teale  Streets 
Culver  City,  California 

Attn:  Dr.  L.C.  Van  Atta,  Associate  Director 
Research  Labs, 

Hycon  Eastern,  Inc. 
75  Cambridge  Parkway 
Cambridge,  Mass. 
Attn:  Mrs,  Lois  Seulowitz 
Technical  Librarian 

International  Business  Machines  Corp, 

Military  Products  Division 

590  Madison  Avenue 

New  York  33,  New  York 

Attn;  Mr.  C.F.  McElwain,  General  Manager 

International  Business  Machines  Corp. 
Military  Products  Division 
Owego,  New  York 

Attn:  Mr,  D,  I.  Marr,  Librarian 
Department  b59 

International  Resistance  Company 
UOl  N.  Broad  Street 
Philadelphia  8,  Pa. 
Attn:  Research  Library 

Jansky  and  Bailey,  Inc. 
1339  Wisconsin  Avenue,  N,  W, 
Washington  7,  D.  C. 
Attnt  Mr.  Dolmer  C.  Ports 

Dr.  Henry  Jaslk,  Consulting  Engineer 
298  Shames  Drive 
Brush  Hollow  Industrial  Park 
Westbury,  New  York 

Electromagnetic  Research  Corporation 
711  Ibth  Stfeet,  N.  W, 
Washington  5,  D.  C, 
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Lockheed  Aircraft  Corporation 

2?55  N.   Hollywood  Way 

California  Dl^dsion  Engineering  Library 

Department  72-75,    Plant  A-1,   Bldg.   63-1 

Purbank,   California 

Attn:      N.   0.    Harnois 

The  Martin  Company 

P.  0.   Box  179 

Denver  1,  Colorado 

Attn:  Mr,  Jack  McCormick 

The  Glenn  L.  Martin  Companv 
Baltimore  3,  Maryland 
Attn:  Engineering  Library 
Antenna  Design  Group 

Maryland  Electronic  Manufacturing  Corp. 
?009  Calvert  Road 
College  Park,  Maryland 
Attn;  Mr.  H.  Warren  Cooper 

Mathematical  Reviews 
190  Hope  Street 
Providence  6,  Rhode  Island 

The  W,  L.  Maxscn  Corporation 

h60  West  ?bth  Street 

New  York,  N.  Y. 

Attn:     Mi3s  Dorothy  Clark 

McDonnell  Aircraft  Corporation 

Lambert  Saint-Louis  Municipal  Airport 

Box  ?16,   St,   Louis   3,   Missouri 

Attn:      R.  D,   Detrich,   Engineering  Library 

McMillan  Laboratory,    Inc. 

Prownville   Avenue 

Ipswich,   Massachusetts 

Attn:     Security  Officer.    Document  Rocrni 


Melpar,  Inc.  (2) 

3000  Arlington  Boulevard 

Falls  Church,  Virginia 

Attn:  Engineering  Technical  Library 

Microwave  Development  Laboratory 
90  Broad  Street 
Babson  Park  57,  Massachusetts 
Attn:  N.  Tucker,  General  Manager 

Microwave  Raiiation  Company  Inc. 

19223  South  Hamilton  Street 

Gardena,  California 

Attn:  Mr.  Morris  J.  Ehrlich,  President 

Chance  Vought  Aircraft,  Ino, 
931b  West  Jefferson  Street 
Dallas,  Texas 
Attn:  Mr.  H.  S,  White,  Librarian 

Northrop  Aircraft,  Inc. 
Hawthorne,  California 

Attn:  Mr.  E.  A.  Freitas,  Library  Dept31'i5 
inoi  E,  Broadway 

Reminpjton  Rand  Univ.  -  Division  of  Sperrv 

Rand  Corporation 
1900  West  Allegheny  Avenue 
Philadelphia  29,  Pennsylvania 
Attn:  Mr.  John  F.  McCarthy 

R  and  D  Sales  and  Contracts 

North  American  Aviation,  Inc. 

1221b  Lakewood  Boulevard 

Downey,  California 

Attn:  Engineering  Library  b95-ll5 

North  American  Aviation,  Inc. 
los  Angeles  Interna ti'^nal  Airport 
Lcs  Angeles  b5,  California 
Attn:   Engineering  Technical  File 

Page  Communications  Engineers,  Inc. 
710  Fourteenth  Street,  Northwest 

Washington  5,  D.  C, 
Attn:  Librarian 

Phllco  Corporation  Research  Division 

Branch  Library 

b700  Wissachickon  Avenue 

Philadelphia  'Jj,  Pa. 

Attn:  Mrs.  Dorothy  S.  Collins 


Plckard  and  Bums,  Inc. 

2bO  Highland  Avenue 

Needham  9b,  Mass. 

Attn:  Dr.  J.  T.  DeBettencourt 

Polytechnic  Research  and  Development 

Company,  Inc. 

202  Tiilary  Street 

Brooklyn  1,  New  York 

Attn:  Technical  Library 

Radiation  Engineering  Laboratory 
Main  Street 
Maynard,  Mass. 
Attn:  Dr.  John  Ruze 

Radiation,  Inc. 

P.  0.  Drawer  37 

Melbourne ,  Florida 

Attn:  Technical  Library,  Mr.  M.L,  Cox 

Radio  Corp,  of  America 

RCA  Laboratories 

Rocky  Point,  New  York 

Attn:   P.  S.  Carter,  Lab.  Library 

RCA  Laboratories 
David  Sarnoff  Research  Center 
Princeton,  New  Jersey 
Attn:   Miss  Fern  Cloak,  Librarian 
Research  Library 

Radio  Corporation  of  America 
Defense  Electronic  Pre  nicts 
Building  10,  Floor  7 
Camden  2,  New  Jeraey 
Attn:  Mr.  Harold  J.  Schrader 

Staff  Engineer,  Organization 

of  Chief  Technical 

Adm-inistrator 

The  Ramo-Wooldridge  Corporation 
P.O.  Box  b<b53  Airnort  Station 
Los  Angeles  b5,  California 
Attn:  Margaret  C.  Whitnah, 
Chief  Librarian 

Hoover  Mic^jwave  Co. 
9592  Baltimore  Avenue 
College  Park,  Maryland 

Director,  USAF  Project  RAND 

Via:  Air  Force  Liaison  Office 

The  Rand  Corporation 

1700  Main  Street 

Santa  Monica,  California 

Rantec  Corporation 

Calflbasas,  California 

Attn:   Grace  Keener,  Office  Manager 

Raytheon  Manufacturing  Comr>any 
Missile  Systems  Division 
Bedford.  Mass. 
Attn:  Mr.  Irving  Goldstein 

Raytheon  Manufacturing  Companv 
Wavland  Laboratory,  State  Road 
Wsvland,  Mass. 
Attn:  Mr.  Robert  Borts 

Raytheon  Manufacturing  Company 
Wavland  Laboratory 
Wayland,  Mass, 

Attn:  Miss  Alice  G.  Anderson, 
Librarian 

Republic  Aviation  Corporation 
Farmingdale,  Long  Island,  N.  Y, 
Attn:  Engineering-  Library 


Thru:  Air  Force  '^lant  Representative 
Republic  Aviation  Corp. 
^amingdale.  Long  Island,  N,Y, 


Ryan  Aeronautical  Comnanv 
Lindbergh  Field 
San  Diego  12,  California 
Attn:  Librar-/  -  unclassified 

"jsge  Laboratories 
159  Linden  Street 
Wallesley  8l,  Mass. 

Sanders  Associates 
95  Canal  Street 
Nashua,  New  Hampshire 
Attn:  N.  R.  Wild,  Library 

Sandia  Corporation,  Sandia  Base 

".0.  Box  5^00,  Albuquerque,  New  Mexico 

Attn:  Classified  Document  Division 

Sperry  Gyroscope  Company 

Great  Neck,  Long  Island,  New  York 

Attn:   Florence  W.  Tumbull,  Engr.  Librarian 

Stanford  Research  Institute 

Menlo  Park,  California 

Attn:  Library,  Engineering  Division 

Sylvania  Electric  Products,  Inc. 

100  Pirst  Avenue 
WalthaTi  Sh,   Mass. 

.'ittn:  Charles  A.  Thornhill,  Report  librarian 
Walthan  Laboratories  Library 

Systems  Laboratories  Corporation 
Hi '^52  Ventura  Boulevard 
Sheman  Oaks,  California 
Attn:  Donald  L.  Margerum 

TR3,  Tnc, 

17  Union  Square  West 

New  York  3,  N.  Y. 

Attn:     M,   L.   Henderson,    Librarian 

A.  S.  Thomas,  Inc. 

161  Devonshire  Street 

Boston  10,  Mass. 

Attn:  A.  S.  Thomas,  President 


Rheem  Manu''acturlng  Conpany 
9236  East  Hall  Hoad 
Downey,  California 
Attn:   J.  C,  Joerger 

Trans-Tech,  Inc. 
P.  0,  Box  3b6 
Frederick,  Maryland 


Pell  Telephone  Laboratories 
Murray  Hill 
New  Jersey 

Chu  Associates 
p.  0.  Box  3^7 
Whitcomb  Avenue 
Littleton,  Mass, 

Microwave  Associates,  Inc. 
Burlington,  Mass. 

Raytheon  Manufacturing  Companv 
Missile  Division 
Hartwell  Road 
Pedford,  Mass. 

Radio  Corporation  of  America 
Aviation  Systems  laboratory 
225  Crescent  Street 
Waltham,  Mass. 

Lockheed  Aircraft  Corporation 
Missile  Systems  Division  Research  Library 
Box  50b,  Sunnyvale,  California 
Attn:  Miss  Eva  Lou  Robertson, 
Chief  Librarian 

The  Rand  Corporation 
1700  Main  Street 
Santa  Monica,  California 
Attn:  Dr.  W.  C.  Hoffman 

Commander 

AF  Office  of  Scientific  Research 

Air  Research  and  Development  Command 

lUth  Street  and  Constitution  Avenue 

Washington,   D,   C, 

Attn:     Mr.  Otting,   SRY 

Westinghouse  Electric  Corp. 
Electronics  Division 
Friendship  Int'l  Airport  Box  7b6 
Baltimore  3,   Marvland 
Attn:     Engineering  Library 
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Wheeler  Laboratories,  Inc. 
12?  Cutter  Mill  Road 
Rreat  Neck,  New  York 
Attn:  Mr.  Harold  A.  Wheeler 

Zenith  Plastics  Co. 
Box  91 

rtardena,  California 
Attn:  Mr.  S.  S.  Oleesky 

Library  Geophysical  Institute 
or  the  Oniversity  of  Alaska 
College 
Alaska 

University  of  California 

Berkeley  U,   California 

Attn:  Dr.  Samuel  Silver, 

Prof,  Engineering  Science 
Division  of  Elec.  Eng.  Electronics 
Research  Lab. 

'Inlversitv  of  California 
Electronics  Hesoarch  Lab. 
332  Cory  Hall 
Berkeley  li,  California 
Attn:  J,  R,  Whinnery 

California  Institute  of  Technology 
Jet  Propulsion  Laboratory 
IjRnO  Oak  Crove  Drive 
Pasadena,  California 
Attn:  Mr.  I.  E.  Newlan 

California  Institute  of  Technology 
1201  E.  Calif  oml  3  Street 
Pasadena,  California 
Attn:  Dr.  C.  Papas 

Carnegie  Institute  of  Technology, 
Schenlev  Park 

Pittsburgh  13,  Pennsylvania 
Attn:  Prof.  A.  E.  Heins 

Cornell  University 

School  of  Electrical  Engineering 

Ithaca,  New  York 

Attn:  Prof.  0.  C.  Dalman 

University  of  Florida 

Department  of  Electrical  Engineering 

Gainesville,  Florida 

Attn:  Prof.  V.   H.  Latour,  Library 

Lib'-ary 

Georgia   Institute  of  Technology 

Engineering  Experiment   Station 

Atlanta,    Georgia 

Attn:     Mrs.    J.H.   Crooland,    Librarian 

Harvard  University 

Technical  Reports  Collection 

Gordon  McKay  Librarv,    303A  Pierce  Hall 

Oxford  Street,    Cambridge  3",    Mass. 

Attn:     Mrs.   E.L.   Hufschmidt,    Librarian 

Harvard  College  Observatory 
60  Garden  Street 
Cambridge  39,    Mass. 
Attn:     Dr.  Fred  L.   Whipple 

University  of   Illinois 
Documents  Division  Library 
Urbana,    Illinois 

University  of  Illinois 
College  of  Engineering 
Urbana,    Illinois 

Attn:     Dr.   P.   E.   Moyes,    Department  of 
Electrical  Engineering 

The  Johns  Hopkins  University 
Homewood  Campus 
Department  of   Physics 
Baltimore  18,   Maryland 
Attn:     Dr.   Donald  E.   Kerr 

Sandia  Corporation 
Attn:     Organization  ll|23 
Sandia  Base 
Albuquerque,    New  Mexico 


Applied  Physics  Laboratory 
The  Johfls Hopkins  University 
8621  Georgia  Avenue 
Silver  Spring,   Maryland 
Attn:     Mr.    George  L.   Selelstad 

Massachusetts  Institute  of  Technology 

Research  Laboratory  of  Electronics 

Room  20B-221 

Canfcridge  39,    Massachusetts 

Attn:      John  H.    Hewitt 

Massachusetts   Institute  of  Technology 

Lincoln  Laboratory 
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